Data Set Information¶
The purpose of the case study is to classify a given silhouette as one of four different types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Attribute Information:
- compactness: (average perim)^2/area
- circularity: (average radius)^2/area
- distance_circularity: area/(av.distance from border)^2
- radius_ratio: (max.rad-min.rad)/av.radius
- pr.axis_aspect_ratio: (minor axis)/(major axis)
- max.length_aspect_ratio: (length perp. max length)/(max length)
- scatter_ratio: (inertia about minor axis)/(inertia about major axis)
- elongatedness: area/(shrink width)^2
- pr.axis_rectangularity: area/(pr.axis length*pr.axis width)
- max.length_rectangularity: area/(max.length*length perp. to this)
- scaled_variance: (2nd order moment about minor axis)/area along major axis
- scaled_variance.1: (2nd order moment about major axis)/area along minor axis
- scaled_radius_of_gyration: (mavar+mivar)/area
- scaled_radius_of_gyration.1
- skewness_about: (3rd order moment about major axis)/sigma_min^3 major axis
- skewness_about.1: (3rd order moment about minor axis)/sigma_maj^3 minor axis
- skewness_about.2
- hollows_ratio: (area of hollows)/(area of bounding polygon)
- class: van, car, bus
Where sigma_maj^2 is the variance along the major axis and sigma_min^2 is the variance along the minor axis, and area of hollows = area of bounding poly-area of object
Importng Libraries¶
In [1]:
# To enable plotting graphs in Jupyter notebook
%matplotlib inline
# Numerical libraries
import numpy as np
from sklearn.model_selection import train_test_split
# Import Linear Regression machine learning library
from sklearn.tree import DecisionTreeRegressor
# to handle data in form of rows and columns
import pandas as pd
# importing ploting libraries
import matplotlib.pyplot as plt
#importing seaborn for statistical plots
import seaborn as sns
#importing svm
from sklearn.svm import SVC
# importing GridSearchcv
from sklearn.model_selection import GridSearchCV
#importing kfold
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
# Feature Engineering
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
#metrics
from sklearn import metrics
from sklearn.metrics import classification_report, roc_auc_score, auc
from sklearn.utils import resample
from sklearn import preprocessing
random_state = 42
np.random.seed(random_state)
# Suppress warnings
import warnings; warnings.filterwarnings('ignore')
Reading Data¶
We read the csv file given into a dataframe
In [2]:
data = pd.read_csv("vehicle-1.csv")
In [3]:
print('The shape of the data is',data.shape)
1. Understanding data - Data Preprocessing¶
In [4]:
data.head()
Out[4]:
In [5]:
data.tail()
Out[5]:
In [6]:
data.info()
As we can see there are some missing values in our dataset and they have to be handled before machine learning algorithm are applied
In [7]:
# Check for any Non-Real value present in the dataset such as '?' or '*' etc.
data[~data.iloc[:,:-1].applymap(np.isreal).all(1)]
Out[7]:
In [8]:
data.isnull().sum()
Out[8]:
Observation:
- There are some missing values and these have to be taken care of before training
- Null values are present in circularity, diatance_circularity, radius_ratio, pr.axis_aspect_ratio, scatter_ratio, elongatedness, pr.axis_rectangularity, scaled_variance, scaled_variance.1, scaled_radius_of_gyration, scaled_radius_of_gyration.1, skewness_about, skewness_about.1, skewness_about.2
1.1 Replacing Null values with Median of that respective attribute¶
In [9]:
nulls = data.isnull().sum()
nulls[nulls > 0]
data.fillna(data.median(), inplace=True)
In [10]:
data.isnull().sum()
Out[10]:
1.2 Looking for Outliers¶
In [11]:
# A quick check to find columns that contain outliers
fig = plt.figure(figsize = (15, 10))
ax = sns.boxplot(data = data.iloc[:, 0:18], orient = 'h')
As we can see, there are outliers
1.3 Counting the number of outliers¶
In [12]:
Q1 = data.quantile(0.25)
Q3 = data.quantile(0.75)
IQR = Q3 - Q1
In [13]:
((data < (Q1 - 1.5 * IQR)) | (data > (Q3 + 1.5 * IQR))).sum()
Out[13]:
1.4 Handling Outliers¶
In [14]:
for idx, col in enumerate(data.columns[:-1]):
q1 = data[col].quantile(0.25)
q3 = data[col].quantile(0.75)
low = q1 - 1.5*(q3 - q1)
high = q3 + 1.5*(q3 - q1)
data = data[(data[col] < high) & (data[col] > low)]
In [15]:
# Boxplot after removing outliers
# A quick check to find columns that contain outliers
fig = plt.figure(figsize = (15, 10))
ax = sns.boxplot(data = data.iloc[:, 0:18], orient = 'h')
- There is one outlier in scaled_variance.1, but this can be ignored because it is just one outlier
In [16]:
data.info()
2. Understanding the Attributes/columns¶
2.1 5 - Point Summary¶
In [17]:
# Five point summary
data.describe().T
Out[17]:
2.2 Univariate Analysis¶
We perform univariate analysis to know the distribution of independent variables.
In [18]:
for i in data.columns:
if i!='class':
print("5 Point Summary of", i, "column is:")
print(data[i].describe().T)
ax = sns.distplot(data[i], color="m")
plt.tight_layout()
plt.show()
print("*"*100)
print(" "*200)
Observation:
- Most of them have two or three peaks and all are approximately normally distributed
- Also, most of them have two clear peaks
- This also indicates that we are dealing with multi class classification
- skewness_about, skewness_about.1 is right skewed
2.3 Bivariate Analysis¶
Bivariate analysis is the analysis of two variables where two variables are analysed to explore the relationship/association between them
Since All of our data is numerical, we can infer results from correlation coefficient.
2.4 Pair Plot¶
In [19]:
sns.pairplot(data, diag_kind='kde', hue = 'class') # to plot density curve instead of histogram
Out[19]:
Observation:
- From diagnol, we can understand why there are peaks before, during univariate distribution
- Many of the independent variables are linearly related either positively or negatively and the strength of corelation can be seen in corelation matrix
2.5 Corelation MAtrix¶
In [20]:
# Visualize the correlation among independent features
from matplotlib import pyplot as plt
plt.figure(figsize=(18,18))
ax = sns.heatmap(data.corr(), vmax=.6, square=True, fmt='.2f', annot=True, linecolor='white', linewidths=0.4)
plt.title('Correlation heatmap')
plt.show()
In [21]:
data.groupby(by=['class'])['compactness'].sum().reset_index().sort_values(['compactness']).tail(10).plot(x='class',y='compactness',kind='bar',figsize=(15,5))
plt.show()
2.6 Jointplots of few weekly corelated attributes / variables¶
As we can see the below plots, we can see that joint plot gives straight line parallel to x axis which indicates a week relation betweent hose two attributes
In [22]:
for i in ['pr.axis_rectangularity', 'max.length_rectangularity',
'scaled_variance', 'scaled_variance.1']:
sns.jointplot(i, 'hollows_ratio', data, kind='reg', size=7, color='r')
In [23]:
sns.jointplot('skewness_about', 'hollows_ratio', data, kind='reg', size=7, color='r')
Out[23]:
In [24]:
sns.jointplot('skewness_about', 'skewness_about.1', data, kind='reg', size=7, color='r')
Out[24]:
In [25]:
sns.jointplot('max.length_rectangularity', 'skewness_about.1', data, kind='reg', size=7, color='r')
Out[25]:
In [26]:
sns.jointplot('scaled_radius_of_gyration.1', 'scatter_ratio', data, kind='reg', size=7, color='r')
Out[26]:
2.6 Jointplots of few high corelated attributes / variables¶
In [27]:
for i in ['skewness_about.2', 'hollows_ratio']:
sns.jointplot(i, 'scaled_radius_of_gyration.1', data, kind='reg', size=7, color='r')
Observation:
- As we can see, elongatedness is important attribute because it gave highly negative correlation with many other attributes
- Most of the variables are related to each other, so we cannot remove any attribute out of the algorithm
In [28]:
for i in ['compactness', 'circularity', 'distance_circularity', 'radius_ratio','pr.axis_rectangularity', 'max.length_rectangularity',
'scaled_variance', 'scaled_variance.1', 'scaled_radius_of_gyration']:
sns.jointplot(i, 'elongatedness', data, kind='reg', size=7, color='r')
Observation:
- Elongatedness is negatively corelated to these attributes - 'compactness', 'circularity', 'distance_circularity', 'radius_ratio','pr.axis_rectangularity', 'max.length_rectangularity','scaled_variance', 'scaled_variance.1', 'scaled_radius_of_gyration'
In [29]:
for i in ['skewness_about.2', 'hollows_ratio']:
sns.jointplot(i, 'scaled_radius_of_gyration.1', data, kind='reg', size=7, color='r')
Observation:
- scaled_radius_of_gyration.1 is negatively corelated to these attributes - 'skewness_about.2', 'hollows_ratio'
2.7 Boxplot and violin plot of attribute vs dependent variable¶
In [30]:
for i in data.columns:
if i!='class':
plt.figure(figsize=(10,4))
plt.title(i+' and class Boxplot')
sns.boxplot(x='class', y=i, data=data,hue='class')
plt.show()
plt.figure(figsize=(10,4))
plt.title(i+' and class Boxplot')
sns.violinplot(x='class', y=i, data=data,hue='class')
plt.show()
Observation:
- As we can see most of these have an overlap between multio classes ( van, car, bus ) and hence no conclusion can be drawn
2.7 Understanding Target Column or Dependent Variable¶
In [31]:
data['class'].value_counts()
Out[31]:
In [32]:
data['class'].value_counts(normalize = True)*100
Out[32]:
In [33]:
# import the pyplot library
import matplotlib.pyplot as plotter
# The slice names of a population distribution pie chart
pieLabels = 'car', 'bus', 'van'
# Population data
populationShare = [410/801, 203/801, 188/801]
figureObject, axesObject = plotter.subplots()
# Draw the pie chart
axesObject.pie(populationShare,labels=pieLabels,autopct='%1.2f',startangle=90)
# Aspect ratio - equal means pie is a circle
axesObject.axis('equal')
plotter.show()
As we can see, the car:van:bus is almost equal to 2:1:1 and we can treat it as a balanced dataset and hence there is no need of balancing
2.8 Feature Importances - choosing carefully which all attributes have to be a part of the analysis¶
We can get feaure importances using Random Forest
Let us apply random forest and get feature importance so that we can decide whether to keep or delete a particular attribute
In [34]:
#import random forest
from sklearn.ensemble import RandomForestClassifier
In [35]:
# Divide the dataset into Input features and Target variables
a = data.drop('class', axis=1)
b = data['class']
rf = RandomForestClassifier().fit(a, b)
f_p = rf.feature_importances_
res = list(zip(a.columns, f_p*100))
for i in res:
print(i)
Observation:
- As we can see no feature has less than 1 percent importance which indicates that every feature contributes to independent variables
- So we tend to keep all the independent attributes
- max.length_aspect_ratio has the highest feature importance
3. Splitting data into Train and Test¶
In [36]:
#Seperating X and y from data dataframe
X = data.drop(['class'], axis = 1)
y = data['class']
In [37]:
#splitting into train and test
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
#standarizing train and test
x_train = StandardScaler().fit_transform(x_train)
x_test = StandardScaler().fit_transform(x_test)
4. Support Vector Machine¶
In [38]:
# defining parameter range
param_grid = {'C': [0.001, 0.01, 0.1, 0.5, 1, 10, 100, 1000, 10000],
'gamma': [10000, 1000, 100, 10, 1, 0.5, 0.1, 0.01, 0.001],
'kernel': ['rbf','linear','poly']}
grid = GridSearchCV(SVC(), param_grid, cv=5,verbose = 0)
# fitting the model for grid search
best_model=grid.fit(x_train, y_train)
# View best hyperparameters
print('Best gamma:', best_model.best_estimator_.get_params()['gamma'])
print('Best C:', best_model.best_estimator_.get_params()['C'])
print('Best kernel:', best_model.best_estimator_.get_params()['kernel'])
best_gamma=best_model.best_estimator_.get_params()['gamma']
best_c=best_model.best_estimator_.get_params()['C']
best_kernel=best_model.best_estimator_.get_params()['kernel']
print("Best Score:", best_model.best_score_)
4.1 Fitting with best parameters¶
In [39]:
svm=SVC(C=best_c,gamma=best_gamma,kernel=best_kernel)
svm.fit(x_train,y_train)
predicted_labels = svm.predict(x_test)
print("Accuracy: ",svm.score(x_test, y_test))
print(" "*100)
In [40]:
#Confusion matrix
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score, recall_score
print(confusion_matrix(y_test, predicted_labels))
print("Accuracy Score:",accuracy_score(y_test, predicted_labels))
In [41]:
target_names = ['car', 'bus','van']
print(" "*100)
print("classification report for test data:")
print(classification_report(y_test, predicted_labels, target_names=target_names))
Observation:
- Accuracy of the best fit model is 98.8 % which indicates that we have did good in our model
- CAr has a very high recall of 1 followed by bus and van
5. K - Fold Cross Validation¶
In [42]:
# we run kfold cross validation with the default parameters
In [43]:
num_folds = 10
seed = 42
kfold = KFold(n_splits=num_folds, random_state=seed)
model = SVC()
model.fit(x_train, y_train)
results = cross_val_score(model, x_test, y_test, cv=kfold)
print(results)
print("max accuracy:", results.max())
print("min accuracy:", results.min())
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
In [44]:
# we run kfold cross validation with the best parameters found before
In [45]:
num_folds = 10
seed = 42
kfold = KFold(n_splits=num_folds, random_state=seed)
model = SVC(C=best_c,gamma=best_gamma,kernel=best_kernel)
model.fit(x_train, y_train)
results = cross_val_score(model, x_test, y_test, cv=kfold)
print(results)
print("max accuracy:", results.max())
print("min accuracy:", results.min())
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
6. PCA¶
In [46]:
from scipy import stats
Xscaled = stats.zscore(X)
pca = PCA(n_components=18)
pca.fit(Xscaled)
Out[46]:
In [47]:
eig_vals, eig_vecs = pca.explained_variance_, pca.components_
Eigen Values
In [48]:
print(pca.explained_variance_)
Eigen Vectors
In [49]:
print(pca.components_)
And the percentage of variation explained by each eigen Vector
In [50]:
print(pca.explained_variance_ratio_)
In [51]:
tot = sum(eig_vals)
var_exp = [( i /tot ) * 100 for i in sorted(eig_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
print("Cumulative Variance Explained", cum_var_exp)
In [52]:
count = 0
sum = 0
for i in var_exp:
if sum<95:
sum = sum + i
count = count + 1
print("no fo components req to cover 95 % variance:", count)
As we can see if we take 7 components, 96.1 percentage variance is explained¶
In [53]:
plt.bar(list(range(1,19)),var_exp,alpha=0.5, align='center')
plt.ylabel('Variation explained')
plt.xlabel('eigen Value')
plt.show()
In [54]:
plt.step(list(range(1,19)),cum_var_exp, where='mid')
plt.ylabel('Cum of variation explained')
plt.xlabel('eigen Value')
plt.show()
In [55]:
# Ploting
plt.figure(figsize = (10,5))
plt.bar(range(1, eig_vals.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, eig_vals.size + 1), cum_var_exp, where = 'mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
Dimensionality Reduction - repeat 3, 4 , 5¶
Now 7 dimensions seems better compared to 18 dimensions. With 7 variables we can explain over 95% of the variation in the original data!
In [56]:
data_new = pd.read_csv("vehicle-1.csv")
nulls = data.isnull().sum()
nulls[nulls > 0]
data_new.fillna(data.median(), inplace=True)
for idx, col in enumerate(data_new.columns[:-1]):
q1 = data_new[col].quantile(0.25)
q3 = data_new[col].quantile(0.75)
low = q1 - 1.5*(q3 - q1)
high = q3 + 1.5*(q3 - q1)
data_new = data_new[(data_new[col] < high) & (data_new[col] > low)]
#Seperating X and y from data dataframe
X = data.drop(['class'], axis = 1)
y = data['class']
x_train_pca, x_test_pca, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
x_train_pca = StandardScaler().fit_transform(x_train_pca)
x_test_pca = StandardScaler().fit_transform(x_test_pca)
In [57]:
pca7 = PCA(n_components=7)
pca7.fit(x_train_pca)
x_train_pca = pca7.transform(x_train_pca)
x_test_pca = pca7.transform(x_test_pca)
In [58]:
print("Eigen vectors: ",pca7.components_)
print("Eigen Values: ",pca7.explained_variance_ratio_)
In [59]:
XScaled = stats.zscore(X)
Xpca7 = pca7.transform(XScaled)
sns.pairplot(pd.DataFrame(Xpca7))
Out[59]:
In [60]:
sns.pairplot(pd.DataFrame(Xpca7),diag_kind='kde')
Out[60]:
Observation:
- Principal Components are not easily intrepretable as they are cloud shaped
7. Support vector Machine using PCA dimensions¶
Support Vector Machine¶
In [61]:
# defining parameter range
param_grid = {'C': [0.001, 0.01, 0.1, 0.5, 1, 10, 100, 1000],
'gamma': [1000, 100, 10, 1, 0.5, 0.1, 0.01, 0.001],
'kernel': ['rbf','linear','poly']}
from sklearn.model_selection import StratifiedKFold
cv = StratifiedKFold(n_splits=3)
grid = GridSearchCV(SVC(), param_grid, cv=cv, verbose = 0)
# fitting the model for grid search
best_model_pca=grid.fit(x_train_pca, y_train)
# View best hyperparameters
print('Best gamma:', best_model_pca.best_estimator_.get_params()['gamma'])
print('Best C:', best_model_pca.best_estimator_.get_params()['C'])
print('Best kernel:', best_model_pca.best_estimator_.get_params()['kernel'])
best_gamma_pca=best_model_pca.best_estimator_.get_params()['gamma']
best_c_pca=best_model_pca.best_estimator_.get_params()['C']
best_kernel_pca=best_model_pca.best_estimator_.get_params()['kernel']
print("Best Score:", best_model_pca.best_score_)
Fitting with best parameters¶
In [62]:
svm_pca=SVC(C=best_c_pca,gamma=best_gamma_pca,kernel=best_kernel_pca)
svm_pca.fit(x_train_pca,y_train)
predicted_labels = svm_pca.predict(x_test_pca)
print("Accuracy: ",svm_pca.score(x_test_pca, y_test))
print(" "*100)
In [63]:
#Confusion matrix
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score, recall_score
print(confusion_matrix(y_test, predicted_labels))
print("Accuracy Score:",accuracy_score(y_test, predicted_labels))
In [64]:
target_names = ['car', 'bus','van']
print(" "*100)
print("classification report for test data:")
print(classification_report(y_test, predicted_labels, target_names=target_names))
K - Fold Cross Validation with PCA Principal Components¶
In [65]:
# we run kfold cross validation with the default parameters
In [66]:
num_folds = 20
seed = 42
kfold = KFold(n_splits=num_folds, random_state=seed)
model = SVC()
model.fit(x_train_pca,y_train)
results = cross_val_score(model, x_test_pca, y_test, cv=kfold)
print(results)
print("max accuracy:", results.max())
print("min accuracy:", results.min())
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
In [67]:
# we run kfold cross validation with the best parameters found before
In [68]:
num_folds = 20
seed = 42
kfold = KFold(n_splits=num_folds, random_state=seed)
model = SVC(C=best_c,gamma=best_gamma,kernel=best_kernel)
model.fit(x_train_pca,y_train)
results = cross_val_score(model, x_test_pca, y_test, cv=kfold)
print(results)
print("max accuracy:", results.max())
print("min accuracy:", results.min())
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
8. Comparing Models¶
In [69]:
from prettytable import PrettyTable
x = PrettyTable()
x.field_names = ["Model", "Data", "Accuracy", "gamma", "c", "kernel"]
x.add_row(["SVM", 'Raw', 98.86,best_gamma, best_c, best_kernel])
x.add_row(["SVM", 'PCA', 93.69,best_gamma_pca, best_c_pca, best_kernel_pca])
print(x)
x = PrettyTable()
x.field_names = ["Model", "Data", "Accuracy", "gamma", "c", "kernel"]
x.add_row(["SVM with 10 fold", 'Raw - Default parameters of SVM', 93.94, best_gamma, best_c, best_kernel])
x.add_row(["SVM with 10 fold", 'Raw - Best parameters of SVM', 95.81,1,1,'rbf'])
x.add_row(["SVM with 10 fold", 'PCA - Default parameters of SVM', 91.07, best_gamma_pca, best_c_pca, best_kernel_pca])
x.add_row(["SVM with 10 fold", 'PCA - Best parameters of SVM', 91.75,1,1,'rbf'])
print(x)
9. Conclusions¶
- Removed around 40 outliers but our SVM models captured a lot of information with the best parameters
- We have caluclated corelation matrix and found which attributes are related or not
- Then we have used Random Forest to get the feature importance and have observed that to keep all the independent attributes
- We have split the data into train and test with a random_state = 42 and got test accuracy of 98.86
- We have done k fold Cross validation where k = 40
- we did PCA and found that 7 principal components are enough to cover 95 % variance
- We have then used these 7 principal Components and applied Svm to get a test accuracy of 93.7
- We got 93.7 % accuracy with only 7 dimensions instead of 98.8 with 18 dimensions
- PCA Components are not intrepretable as these are linear combinations of raw data